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Artificial Intelligence & Eva 
Top VPN questions December 9, 2020 | 11:00-12:00 p.m. 


Working remo 


> Mike Amato, Olive Kilpatrick, Jeff Martinson & Justin Stiles 


| 
| 
| | have a question about Office 365 
| 


So, how do | ask Eva a question? | 


AGENDA «learn 


Online learning and resources for I+IT staff. 


Time Topic Presenter 


Monica Sharma, Senior Manager, OCCIO, TORU, MGCS 


11:00-11:10 nd 
Mike Amato, Director, Process and Operations Branch, ITS 
Eva background, Mike Amato, Director, Process and Operations Branch, ITS 
11:10-11:55 development, Justin Stiles, Sr. Manager, Service Operations, ITS 
i i improvements and | Jeff Martinson, Manager, OPS IT Service Desk, ITS 
questions Olive Kilpatrick, I&IT Service Operations Team Lead, ITS 


11:55-12:00 Closing Remarks Monica Sharma, Senior Manager, OCCIO, TORU, MGCS 
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What is a chatbot? 


Chatbots are software that can 
perform automated tasks, such 
as answering Frequently Asked 
Questions or simple requests. 


Chatbots live online in chat 
platforms and usually improve 
over time by using artificial 
intelligence (Al) and machine 
learning. 
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Beginning the Chatbot Journey with LiveChat 


LiveChat is a digital initiative that was piloted in the OPS. 
LiveChat allows two-way interaction between OPS customers 
and the OPS IT Service Desk. 


GOAL: Provide an additional channel of support to OPS 
employees who are seeking assistance from the OPS IT Service 
Desk. 


e 10K+ 


Client satisfaction survey, 97% 
reported overall ease of use. 


Greater than 10,000+ 
LiveChats from the pilot 


group. 
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D LiveChat - Google Chrome - o x 
@ https://intra.myit.esm.tbs.gov.on.ca/ONRequest/Po... € 
L LiveChat [E 


Welcome to the new LiveChat! 


To start the conversation, type your message 
below. 


After you enter your message, it will appear 
here. Feel free to continue typing as we 
connect you with the appropriate service desk 
agent. 


Hi! 


Ir ype your message here 


View of pilot LiveChat window 


Ontario Q 
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Introducing Eva, the Enterprise Virtual Assistant 


In April 2020, the Ontario Public Service (OPS) introduced Eva, an artificial 
intelligence (Al) chatbot to help support the OPS IT Service Desk. By 
Adding Al to our Service Desk, we've automated frequently asked 
questions/issues and freed up Service Desk Agents for higher priority calls. 


In November 2020, Eva was augmented to support Human Resources (HR)/ 
Labour Relations (LR) frequently asked questions. Once launched, new 
content will be added on an ongoing basis. 


10.3K } e 


10.3K+ interactions Reducing load on service Has assisted with 
with staff since April desk while still providing a ` employee transition to 
2020 with a 30.4% call, great user experience for | remote work as a result 
deflection rate users who need support of COVID-19 
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Eva 
Virtual Assistant 


meg 


Eva 14:23 


Welcome back Jeff! | am Eva, your Enterprise Virtual 


Assistant. 


What can | help you with today? Select one of the 


common topics, or type your question below. 
If | dont know the answer, | will connect you with a 
Service Desk Analyst. 


Top VPN questions 


I have a question about Office 365 


| 
| Working remotely 
| 
| 


So, how do | ask Eva a question? 


[Type your message here 


Ontario Q 
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Creating a Chatbot 


User Experience Design 
» Identify Target Audience 


* Accessibility 
* Persona Development N 
* Content Management i 

I 


Research & Development 
Managed Service vs In-House Team 


* Vendor Selection 


I 
I 
I 
I 
i 
Bot Development ! 
* What content to include ! 
. H 
| 
I 
o 


Concepts 
* Demo 4 
I 


Key Consultations 


* Human Resources > 
+ Diversity & Inclusion 5 
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Pilot Group Testing 


* Internal 
+ Engage with power users 


Product Launch 
» Release Strategy 
* Communications Plan 


Measurement Ongoing Improvements 
+ Establish KPIs 
* Reporting 


Ongoing Improvements 
+ Conversation Reviews 
+ New Content 
* Growth 
+ Lessons Learned 


Ontario Q 
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D User Experience Design 


Decisions to Make 


+ Formal or Informal? 

* Official or Casual? 

+ Male or Female? 

* Neutral or Robot? 

+ Friendly or Professional? 

+ How many words is too many? 

* Immediate or paused responses? 
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Things to Think About 


Who are the intended customers? 
How is the service provided now? 
What are the existing 
guidance/rules 

AODA Compliance 


Consultations 


MGCS Communications 
Diversity & Inclusion Office 
Ontario Digital Services 
Ontario Shared Services 
Ministry of Transportation 
Ministry of Finance 
ServiceOntario 

Enterprise Digital Services 
Integration Division 
Ministry of Training, Colleges and 
Universities 


Ontario Q 
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“q 


ae User Experience Design (continued) 


Ef TONYA 
What's in a name? IA ABBOT 


* Over 100 names were considered for our og 


chatbot 
CODEEL.I.S.A 
+ A gender-neutral name was agreed upon ST "80803 


(Onya), which was later abandoned after N 
feedback from various stakeholders è rad <= CH YS 


uu SAKON SD. SS ONT 
+ Eva (Enterprise Virtual Assistant) Was agreed OPIE ITBOTEHATNOWISAAC 


after multiple "naming workshops" and 


finalized after our partner groups gave their ODIEWALTON 
feedback & end t 
eedbac endorsemen SIVAJESSA 
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© User Experience Design (continued) 


Create a bio for your chatbot: 


e What is their role in the 
organization? 


e What is their personality? 


e What is their style? 


e What are their job goals? 


e How will they respond to 
questions? 
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Personality Spectrum 


Relate-able Brand 


x. 


SIMPLE 


Ontario Q 
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2 Research & Development 


A? 


Decisions to Make 


Vendor Selection 
* Consulted with Gartner (Magic Quadrant) 
* Demo of IPsoft's Amelia 
e Tried Microsoft QnA Maker 


Managed Service vs In-House Development 
* Proof of Concept managed service 
e Evaluated and decided to go with in-house 
team trained by IBM 
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Things to Think About 


Who are your users? 

Is there research on best practices for build? 
Accessibility? 

What are the intended benefits? 


Ontario 
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@) Chatbot Development - What content to include 
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Analysis of call volume 
e What topics drive the most calls 


Evaluation 
e Which topics are suitable for the bot? 
Don't require an agent to answer 
o We have answers for 
o Answers are short 
o Requires agent, but we can gather upfront 
data to speed up contact 


(e) 


Automation opportunities 
* What do we have access to automate 
e Can we get access? 


Ontario Q 
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© Bot Development - Concepts a 


MYTH: Load up the bot with information and you are done! 


New Concepts: 
auto Lemina bota feature 
* The chatbot interprets questions asked with * The chatbot learns from disambiguation 
high confidence to two possible answers choices over time 
+ Controlled by the bot * Increases confidence for one answer over the 


other 
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5 Pilot Groups & Testing 


Various groups included in the testing process: 


IT: 
e Internal team leads and managers 
* Broader ITS community 


Non-IT: 
* Office 365 Champions 
* 2,500 existing LiveChat users 


Testing was primarily done to ensure the chatbot could 
give the answers we expected. 


Bonus: Also provided early insights into user behaviour 
and different ways of asking the same question. 
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o O 


Security Personality 
Navigation / Onboarding 
Error Management Intelligence 


© © 


Speed Accuracy 


Ontario Q 
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(6) Launch 
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Have a plan 

Plan, plan & plan some more, but be prepared to 
change at the last minute — you never know when 
a pandemic might strike. 


Be flexible 

Eva was originally going to launch ministry by 
ministry over a period of several months, but 

quickly shifted to a "big bang” launch to help 

support the quick shift to remote work. 


Get the word out 

Communicate as much as possible, in as many way 
you can — more visibility means more interactions! 
(e.g. OPS Weekly, IT Service Announcement, OPS 
Working Remotely page, etc.) 


Ontario Q 
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© Measurement 


11/13/2020 11:15:03 AM 


| Livechat & 
Enterprise Eva Engagements 
) x =- og 


= 17,731 


Deflected Rate 


há 2020 Sep 2020 


CSAT Trending 
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Bot Movements by Year, Month and Attribute 


Attribute ®DS @HRLR O 


Bot Movements 


6K 


Y 


OK 


2020 June 


September 


Date Month 


nistries O Mobile @0365 BOSS @ Other 


October 


November 
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Child Bots 
DS 
HRLR 
Ministries 
Mobile 
0365 
oss 
Other 
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Ongoing Improvement: 


Conversation Reviews & New Content 


Conversation Reviews 


* The key to maintaining the bot is keeping record of 
what's been trained 


* The key to improvement is reviewing use of the bot 


21 
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261 


The ability to add new content quickly and target 
the current “hot topic” is critical (e.g. COVID) 


Eva 11:1 
Select from the list. or ask your own question. 


Eva 11:17 
Welcome back Olive! | am Eva, your Enterprise 
Virtual Assistant 


What can help you with today? Select one of the I need to order VPN 
common topics, or type your question below. 
If | dont know the answer, | will connect you with a 


Service Desk Analyst 


Top VPN questions Lost/replacement VPN token 


VPN transfer to someone else 


Working remotely 


( ) 
( How much does VPN cost? ) 
[| Does this user have VPN account? ] 
( ) 
| ) 
| ) 


See more VPN topics 


J 
I have a question about Office 365 ) 
J 


So, how do | ask Eva a question? 


Ontario Q 
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(8) Ongoing Improvement: Growth 
“Chatbots can have a finite HE Eve - Parent / Child bots 


amount of content before 


becoming less efficient” 


e Through the pilot phase, initial 
launches and full release to 
the OPS (April 8, 2020), Eva 
was a single bot 


* In June 2020, we divided the 
content of Eva into three child 
bots, based on volume of 


content per subject area & é 


E Child bots 


e In August 2020, we added a 
fourth child bot for Mobile IT 


Office 365 bot — content about MS Office 365 

Ministry / Cluster bot - content about Ministry specific applications; EG: SAMS 

Mobile IT bot - content about mobile phones; EG: ordering, setting up, checking account. 

DS/Other bot — content about Desktop SW (Windows) and all other content. Content 
HR/ LR bot - Human Resources, Labour Relations and Benefits content. 

Future OSS bots — OSS content such as IFIS, Pay and Benefits, Travel Card. 


* The end user still has the 
experience of a single bot 
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Lessons Learned 


Your chatbot will get Be confident in Give your chatbot high 
stumped - updates area your design & choose a visibility for increased 
constant and ongoing visual identity that will interactions — how do 
process resonate with your you nudge people to 
audience leverage this channel? 
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Technology 
continues to move 
forward at a rapid 
pace — be ready to 
evolve and adapt 


Ontario Q 


Eva Demo 


Introducing Eva, your Enterprise Virtual Assistant. 
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Content Demo 
Adding new content to 
your chatbot 
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Try it yourself - Build your own chatbot! 


If you're interested, you can try building a chatbot bot for free: 


Demo: 


https://watson-conversation-duo-dev.ng.bluemix.net/ 


Build: 


https://www.ibm.com/cloud/garage/dte/producttour/create- 
chatbot-plan-trip 


Documentation: 
https://cloud.ibm.com/docs/services/assistant?topic=assistant- 


getting-started 
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learn 


Online learning and resources for I+IT staff. 


THANK YOU 


We hope you enjoyed today's session. 


Look out for the LearnIT Resources newsletter sent to all I+IT staff 
bi-monthly for upcoming learning and development opportunities. 


The session survey link will be emailed to all participants and it is 
also available in the chat. 


LearniT Deep Dive Learning Session: AI - Eva 


